Papers by Greg Ver Steeg
Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuning (2023.findings-acl)
Copied to clipboard
| Challenge: | Efficient finetuning of pretrained language transformers requires a large number of tunable parameters. |
| Approach: | They propose a language transformer finetuning strategy that introduces task-specific parameters in multiple transformer layers. |
| Outcome: | The proposed method outperforms other methods with 4,100 parameters on GLUE tasks with 5% of full finetuning performance. |
Asymmetric Bias in Text-to-Image Generation with Adversarial Attacks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on adversarial attacks against text-to-image models have not explored the reasons for their effectiveness. |
| Approach: | They propose an adversarial attack objective - entity swapping using adversarials and two gradient-based attack algorithms. |
| Outcome: | The proposed attack objective - entity swapping - is easier to replace “human” with “robot” in the prompt “a human dancing in the rain.” |
Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Parameter-efficient tuning (PET) methods fit pre-trained language models to downstream tasks by computing a small compressed update for a subset of parameter updates or appending and fine-tuning a few new parameters to the pre-train network. |
| Approach: | They propose a parameter-efficient tuning method that prunes PET architectures using structured pruning. |
| Outcome: | The proposed method performs well in practice but could be improved via NAS . it can be used to fine-tune large pre-trained language models on multiple tasks . |
Prompt Perturbation Consistency Learning for Robust Language Models (2024.findings-eacl)
Copied to clipboard
Yao Qiang, Subhrangshu Nandi, Ninareh Mehrabi, Greg Ver Steeg, Anoop Kumar, Anna Rumshisky, Aram Galstyan
| Challenge: | Large language models have demonstrated impressive performance on a number of natural language processing tasks, such as question answering and text summarization. |
| Approach: | They propose a method to reduce the performance drop of large language models by regularizing the divergence between losses from clean and perturbed samples. |
| Outcome: | The proposed approach recovers on average 59% and 69% of the performance drop for IC and SF tasks while using ten times fewer augmented data samples. |
Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversal (2022.findings-acl)
Copied to clipboard
Umang Gupta, Jwala Dhamala, Varun Kumar, Apurv Verma, Yada Pruksachatkun, Satyapriya Krishna, Rahul Gupta, Kai-Wei Chang, Greg Ver Steeg, Aram Galstyan
| Challenge: | Language models excel at generating coherent text, but can be biased in multiple ways, including the unfounded association of male and female genders with gender-neutral professions. |
| Approach: | They propose to modify teacher probabilities and augment the training set to learn a fair model during knowledge distillation by modifying teacher probability and augmenting the training sets. |
| Outcome: | The proposed approach reduces gender disparity in open-ended text generated from the distilled and finetuned models with only a minor compromise in utility. |
Nearly-Unsupervised Hashcode Representations for Biomedical Relation Extraction (D19-1)
Copied to clipboard
| Challenge: | Recent studies have shown that locality sensitive hashcodes are useful for biomedical relation extraction tasks. |
| Approach: | They propose to optimize locality sensitive hashcode representations in a nearly unsupervised manner . they use only data points, but not their class labels, for learning . |
| Outcome: | The proposed approach improves accuracy from training to test sets, and the data points are only used for learning . |
StATIK: Structure and Text for Inductive Knowledge Graph Completion (2022.findings-naacl)
Copied to clipboard
Elan Markowitz, Keshav Balasubramanian, Mehrnoosh Mirtaheri, Murali Annavaram, Aram Galstyan, Greg Ver Steeg
| Challenge: | Knowledge graphs (KGs) represent incomplete knowledge bases. |
| Approach: | They propose to use language models to extract semantic information from text descriptions while using Message Passing Neural Networks to capture structural information. |
| Outcome: | The proposed model achieves state of the art on three challenging inductive baselines. |
Measuring and Mitigating Local Instability in Deep Neural Networks (2023.findings-acl)
Copied to clipboard
| Challenge: | Uncertain details like random initialization can change the outputs of a trained system with potentially disastrous consequences. |
| Approach: | They propose a model stability problem by studying how the predictions of a deep neural network change as a consequence of stochasticity in the training process. |
| Outcome: | The proposed method outperforms data-agnostic methods and is 90% cheaper than the gold standard. |
Temporal Generalization for Spoken Language Understanding (2022.naacl-industry)
Copied to clipboard
| Challenge: | Spoken Language Understanding models are usually trained offline on historical data, but must perform well on incoming user requests after deployment. |
| Approach: | They propose different strategies for achieving good temporal generalization . they focus on temporal drift, where the distribution of utterances may change . |
| Outcome: | The proposed model can perform well on unseen domains, e.g., upcoming data. |